System and method for interplay between elements of an augmented reality environment

ABSTRACT

An augmented reality system is disclosed. The system receives values of parameters of real-world elements of an augmented reality environment from various sensors and creates a three-dimensional textual matrix of sensor representation of a real environment world based on the parameters. The system determines a context of a specific virtual object with respect to the real-world environment based on the three-dimensional textual matrix. The system then models the specific virtual object based on the context to place the specific virtual object in the augmented reality environment.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Indian Patent Application No.202111030777 filed on Jul. 8, 2021, the entire contents of which arehereby incorporated by reference.

TECHNICAL FIELD

The present disclosure relates to the field of interactive virtual,augmented or mixed reality environments. More particularly, the presentdisclosure relates to systems and methods for viewing interplay betweenreal-world elements and virtual objects (including interplay betweenvarious virtual objects themselves) of an augmented reality environment.

BACKGROUND

Augmented reality systems provide an interactive experience of areal-world environment where objects that reside in the real-world arealtered by computer-generated perceptual information. Faster computerprocessors of modern computing devices have made it feasible to combinecomputer-generated information with real-time video captured using acamera. In recent years, augmented reality has become more pervasiveamong a wide range of applications including its use in gaming,education, business, repair and maintenance, navigation, entertainment,medicine, and hospitality and tourism, among others.

However, existing systems are generally based on including virtualobjects (e.g. text, graphics, symbols, images, and the likes) in animage of a real-world location. These existing systems that merge thevirtual objects with the real-world, are restricted to live viewing oflimited virtual elements that can have limited interaction with thereal-world elements, interaction with the other virtual elements, andinterplay. There is therefore a need in the art for systems and methodsthat can facilitate simultaneous viewing and interaction of virtualelements in real-world environments that interplay with each other inreal-world context, which overcomes above-mentioned and otherlimitations of existing approaches.

SUMMARY

This summary is provided to introduce simplified concepts of systems andmethods disclosed herein, which are further described below in theDetailed Description. This summary is not intended to identify key oressential features of the claimed subject matter, nor is it intended foruse in determining/limiting the scope of the claimed subject matter.

The present disclosure relates to the field of interactive virtual,augmented or mixed reality environments. More particularly, the presentdisclosure relates to systems and methods for viewing interplay betweenreal-world elements and virtual objects (including interplay betweenvarious virtual objects themselves) of an augmented reality environment.

An aspect of the present disclosure provides a system implemented in acomputing device for viewing interplay between one or more real-worldelements and one or more virtual objects of an augmented realityenvironment. The system comprises an input unit comprising one or moresensors coupled with the computing device, wherein the one or moresensors capture one or more parameters of the one or more real-worldelements of the augmented reality environment; a processing unitcomprising a processor coupled with a memory, the memory storinginstructions executable by the processor to: receiving values of the oneor more parameters from the input unit and creating a three-dimensionaltextual matrix of sensor representation of a real environment worldbased on the one or more parameters; determining a context of a specificvirtual object of the one or more virtual objects with respect to thereal-world environment based on the three-dimensional textual matrix;and modelling the specific virtual object based on the context to placethe specific virtual object in the augmented reality environment.

According to an embodiment, the one or more parameters includeinformation pertaining to any or a combination of light, sound, surface,size, direction of light and physics of the one or more real-worldelements.

According to an embodiment, the one or more sensors include any or acombination of an image sensor, a camera, an accelerometer, a soundsensor and a gyroscope.

According to an embodiment, the three-dimensional textual matrixincludes information pertaining to any or a combination of spaceestimation, light estimation, sound estimation, surface estimation,environment estimation, size estimation, direction of light estimation,and physics estimation with respect to the one or more real-worldelements.

According to an embodiment, the context of the specific virtual objectis determined based on the context of one or more other virtual objects.

According to an embodiment, each virtual object of the one or morevirtual objects is divided into a plurality of parts and subparts.

According to an embodiment, information of the three-dimensional textualmatrix is reusable.

According to an embodiment, the processing unit creates athree-dimensional grid indicating spatial structuring of a user anduser's environment.

According to an embodiment, the three-dimensional grid includes aplurality of cells, and information pertaining to each cell of theplurality of cells is saved in the three-dimensional textual matrix.

Another aspect of the present disclosure provides a method for viewinginterplay between one or more real-world elements and one or morevirtual objects of an augmented reality environment, carried outaccording to instructions saved in a computer device, comprising:receiving values of one or more parameters of the one or more real-worldelements of the augmented reality environment, from an input unit, theinput unit comprising one or more one or more sensors coupled with thecomputing device; creating a three-dimensional textual matrix of sensorrepresentation of a real environment world based on the one or moreparameters; determining a context of a specific virtual object of theone or more virtual objects with respect to the real-world environmentbased on the three-dimensional textual matrix; and modelling thespecific virtual object based on the context to place the specificvirtual object in the augmented reality environment.

Another aspect of the present disclosure provides a computer programproduct, comprising: a non-transitory computer readable storage mediumcomprising computer readable program code embodied in the medium that isexecutable by one or more processors of a computing device to viewinterplay between one or more real-world elements and one or morevirtual objects of an augmented reality environment, wherein the one ormore processors perform: receiving values of one or more parameters ofthe one or more real-world elements of the augmented realityenvironment, from an input unit, the input unit comprising one or moreone or more sensors coupled with the computing device; creating athree-dimensional textual matrix of sensor representation of a realenvironment world based on the one or more parameters; determining acontext of a specific virtual object of the one or more virtual objectswith respect to the real-world environment based on thethree-dimensional textual matrix; and modelling the specific virtualobject based on the context to place said specific virtual object in theaugmented reality environment.

Various objects, features, aspects and advantages of the presentdisclosure will become more apparent from the following detaileddescription of preferred embodiments, along with the accompanyingdrawing figures in which numerals represent like features.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings are included to provide a furtherunderstanding of the present disclosure, and are incorporated in andconstitute a part of this specification. The drawings illustrateexemplary embodiments of the present disclosure and, together with thedescription, serve to explain the principles of the present disclosure.The diagrams are for illustration only, which thus is not a limitationof the present disclosure.

FIG. 1 illustrates architecture of an augmented reality system toillustrate its overall working in accordance with an embodiment of thepresent disclosure.

FIG. 2 illustrates exemplary functional components of a processing unit106 in accordance with an embodiment of the present disclosure.

FIG. 3 illustrates exemplary representations of determination of textualinformation of the three-dimensional matrix in accordance with anembodiment of the present disclosure.

FIG. 4 illustrates exemplary representations of working of the augmentedreality system in accordance with an embodiment of the presentdisclosure.

FIGS. 5A-C illustrate exemplary representations indicating an effect ofchange in parameters on the augmented reality environment in accordancewith an embodiment of the present disclosure.

FIG. 6 illustrates formation of a three-dimensional grid in accordancewith an embodiment of the present disclosure.

FIG. 7 is a flow diagram illustrating a method for viewing interplaybetween real-world elements and virtual objects of an augmented realityenvironment in accordance with an embodiment of the present disclosure.

FIGS. 8A-B are flow diagrams illustrating exemplary working of theaugmented reality system in accordance with an embodiment of the presentdisclosure.

FIG. 9 illustrates an exemplary computer system in which or with whichembodiments of the present disclosure can be utilized in accordance withembodiments of the present disclosure.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth inorder to provide a thorough understanding of embodiments of the presentinvention. It will be apparent to one skilled in the art thatembodiments of the present invention may be practiced without some ofthese specific details.

Various methods described herein may be practiced by combining one ormore machine-readable storage media containing the code according to thepresent invention with appropriate standard computer hardware to executethe code contained therein. An apparatus for practicing variousembodiments of the present invention may involve one or more computers(or one or more processors within a single computer) and storage systemscontaining or having network access to computer program(s) coded inaccordance with various methods described herein, and the method stepsof the invention could be accomplished by modules, routines,subroutines, or subparts of a computer program product.

As used in the description herein and throughout the claims that follow,the meaning of “a,” “an,” and “the” includes plural reference unless thecontext clearly dictates otherwise. Also, as used in the descriptionherein, the meaning of“in” includes “in” and “on” unless the contextclearly dictates otherwise.

Embodiments of the present invention may be provided as a computerprogram product, which may include a machine-readable storage mediumtangibly embodying thereon instructions, which may be used to program acomputer (or other electronic devices) to perform a process. The term“machine-readable storage medium” or “computer-readable storage medium”includes, but is not limited to, fixed (hard) drives, magnetic tape,floppy diskettes, optical disks, compact disc read-only memories(CD-ROMs), and magneto-optical disks, semiconductor memories, such asROMs, PROMs, random access memories (RAMs), programmable read-onlymemories (PROMs), erasable PROMs (EPROMs), electrically erasable PROMs(EEPROMs), flash memory, magnetic or optical cards, or other type ofmedia/machine-readable medium suitable for storing electronicinstructions (e.g., computer programming code, such as software orfirmware). A machine-readable medium may include a non-transitory mediumin which data may be saved and that does not include carrier wavesand/or transitory electronic signals propagating wirelessly or overwired connections. Examples of a non-transitory medium may include, butare not limited to, a magnetic disk or tape, optical storage media suchas compact disk (CD) or digital versatile disk (DVD), flash memory,memory or memory devices. A computer-program product may include codeand/or machine-executable instructions that may represent a procedure, afunction, a subprogram, a program, a routine, a subroutine, a module, asoftware package, a class, or any combination of instructions, datastructures, or program statements. A code segment may be coupled toanother code segment or a hardware circuit by passing and/or receivinginformation, data, arguments, parameters, or memory contents.Information, arguments, parameters, data, etc. may be passed, forwarded,or transmitted via any suitable means including memory sharing, messagepassing, token passing, network transmission, etc.

Various terms as used herein are shown below. To the extent a term usedin a claim is not defined below, it should be given the broadestdefinition persons in the pertinent art have given that term asreflected in printed publications and issued patents at the time offiling.

The present disclosure relates to the field of interactive virtual,augmented or mixed reality environments. More particularly, the presentdisclosure relates to systems and methods for viewing interplay betweenreal-world elements and virtual elements (including interplay betweenvarious virtual objects themselves) of an augmented reality environment.

An aspect of the present disclosure provides a system implemented in acomputing device for viewing interplay between one or more real-worldelements and one or more virtual objects of an augmented realityenvironment. The system comprises an input unit comprising one or moresensors coupled with the computing device, wherein the one or moresensors capture one or more parameters of the one or more real-worldelements of the augmented reality environment; a processing unitcomprising a processor coupled with a memory, the memory storinginstructions executable by the processor to: receiving values of the oneor more parameters from the input unit and creating a three-dimensionaltextual matrix of sensor representation of a real environment worldbased on the one or more parameters; determining a context of a specificvirtual object of the one or more virtual objects with respect to thereal-world environment based on the three-dimensional textual matrix;and modelling the specific virtual object based on the context to placethe specific virtual object in the augmented reality environment.

According to an embodiment, the one or more parameters includeinformation pertaining to any or a combination of light, sound, surface,size, direction of light and physics of the one or more real-worldelements.

According to an embodiment, the one or more sensors include any or acombination of an image sensor, a camera, an accelerometer, a soundsensor and a gyroscope.

According to an embodiment, the three-dimensional textual matrixincludes information pertaining to any or a combination of spaceestimation, light estimation, sound estimation, surface estimation,environment estimation, size estimation, direction of light estimation,and physics estimation with respect to the one or more real-worldelements.

According to an embodiment, the context of the specific virtual objectis determined based on the context of one or more other virtual objects.

According to an embodiment, each virtual object of the one or morevirtual objects is divided into a plurality of parts and subparts.

According to an embodiment, information of the three-dimensional textualmatrix is reusable.

According to an embodiment, the processing unit creates athree-dimensional grid indicating spatial structuring of a user.

According to an embodiment, the three-dimensional grid includes aplurality of cells, and wherein information pertaining to each cell ofthe plurality of cells is saved in the three-dimensional textual matrix.

Another aspect of the present disclosure provides a method for viewinginterplay between one or more real-world elements and one or morevirtual objects of an augmented reality environment, carried outaccording to instructions saved in a computer device, comprising:receiving values of one or more parameters of the one or more real-worldelements of the augmented reality environment, from the input unit, theinput unit comprising one or more one or more sensors coupled with thecomputing device; creating a three-dimensional textual matrix of sensorrepresentation of a real environment world based on the one or moreparameters; determining a context of a specific virtual object of theone or more virtual objects with respect to the real-world environmentbased on the three-dimensional textual matrix; and modelling thespecific virtual object based on the context to place the specificvirtual object in the augmented reality environment.

Another aspect of the present disclosure provides a computer programproduct, comprising: a non-transitory computer readable storage mediumcomprising computer readable program code embodied in the medium that isexecutable by one or more processors of a computing device to viewinterplay between one or more real-world elements and one or morevirtual objects of an augmented reality environment, wherein the one ormore processors perform: receiving values of one or more parameters ofthe one or more real-world elements of the augmented realityenvironment, from an input unit, the input unit comprising one or moreone or more sensors coupled with the computing device; creating athree-dimensional textual matrix of sensor representation of a realenvironment world based on the one or more parameters; determining acontext of a specific virtual object of the one or more virtual objectswith respect to the real-world environment based on thethree-dimensional textual matrix; and modelling the specific virtualobject based on the context to place said specific virtual object in theaugmented reality environment.

Embodiments of the present disclosure aim to provide an augmentedreality system that incorporates a combination of real and virtualworlds, real-time interaction, and accurate three-dimensionalregistration of virtual and real objects. Various embodiments discloseaugmenting and interacting with a number of products in real-time usinga camera-based computing device through a system incorporating varioussensors and functionalities. A contextual three-dimensional (3D)environment and a 3D matrix map is created at runtime using camerainput, real-world input, and virtual input. Each virtual object can beindependently modelled, textured, loaded and compressed as a unique 3Dmodel format, where each virtual object can interplay (materials,vertices, faces, and animations are shared whenever required/optimal)with each other and the real-world objects. The virtual objects can alsointeract with the real world because of real-world detection throughsensory data (e.g. light, sound, textures, reflections, wind, weather,daytime, physics, motion, etc). Interactions and interplay is madepossible using a Graphical User Interface and first-person navigation(e.g. device position, orientation, pose, etc).

FIG. 1 illustrates architecture of an augmented reality system 100 toillustrate its overall working in accordance with an embodiment of thepresent disclosure.

According to an embodiment, an augmented reality system 100(interchangeably referred to as system 100, hereinafter) is implementedin a computing device. The system 100 comprises an input unit 104, aprocessing unit 106 and an output unit 108. The input unit 104 maycomprise one or more pre-processors, which processes perception inputs,raw sensed inputs from sensors of a sensor unit 102 coupled with thesystem 100 including, but not limited to, an image sensor, a camera, anaccelerometer, a gyroscope, and the like. The pre-processed sensedinputs may comprise parameters of elements in real-world environmentsincluding information pertaining to light, sound, surface, size,direction of light, physics, etc. of the real-world elements.

According to an embodiment, the processing unit 106 may comprise aprocessor and a memory and/or may be integrated with existing systemsand controls of the computing device. For instance, signals generated bythe processing unit 106 may be sent to an output unit 108 or a GraphicalUser Interface (GUI) or a display unit of the computing device. Theoutput unit 108 may also be a display device or any other audio-visualdevice.

According to an embodiment, during real-world information processing110, the processing unit 106 receives values of the parameters from theinput unit 102 and creates a 3D textual matrix of sensor representationof a real environment world based on the parameters. The parameters caninclude information pertaining to information pertaining to any or acombination of light, sound, surface, size, direction of light, physics,and the like of the real-world elements such that the 3D textual matrixincludes information pertaining to any or a combination of spaceestimation, light estimation, sound estimation, surface estimation,environment estimation, size estimation, direction of light estimation,physics estimation and the like with respect to the real-world elements.In an implementation, information of the 3D textual matrix can bereused. The processing unit 106 can create a 3D grid indicating spatialstructuring of a user and user's environment such that the 3D gridincludes a plurality of cells, and information pertaining to each cellcan be saved in the 3D textual matrix.

According to an embodiment, during context determination 112, theprocessing unit 106 determines a context of a specific virtual objectwith respect to the real-world environment based on the 3D textualmatrix. Additionally, the context of the specific virtual object can bedetermined based on the context of other virtual objects.

According to an embodiment, during virtual object modelling 116, theprocessing unit 106 models the specific virtual object based on thecontext to place the specific virtual object in the augmented realityenvironment. For effective modelling, each virtual object can be dividedinto a plurality of parts and subparts.

FIG. 2 illustrates exemplary functional components of a processing unit106 in accordance with an embodiment of the present disclosure.

In an aspect, the processing unit 106 may comprise one or moreprocessor(s) 202. The one or more processor(s) 202 may be implemented asone or more microprocessors, microcomputers, microcontrollers, digitalsignal processors, central processing units, logic circuitries, and/orany devices that manipulate data based on operational instructions.Among other capabilities, the one or more processor(s)202 are configuredto fetch and execute computer-readable instructions saved in a memory206 of the processing unit 106. The memory 206 may store one or morecomputer-readable instructions or routines, which may be fetched andexecuted to create or share the data units over a network service. Thememory 206 may comprise any non-transitory storage device including, forexample, volatile memory such as RAM, or non-volatile memory such asEPROM, flash memory, and the like.

The processing unit 106 may also comprise an interface(s) 204. Theinterface(s) 204 may comprise a variety of interfaces, for example,interfaces for data input and output devices, referred to as I/Odevices, storage devices, and the like. The interface(s) 204 mayfacilitate communication of processing unit 106 with various devicescoupled to the processing unit 106 such as the input unit 104 and theoutput unit 108. The interface(s) 204 may also provide a communicationpathway for one or more components of the processing unit 106. Examplesof such components include, but are not limited to, processing engine(s)208 and data 210.

The processing engine(s)208 may be implemented as a combination ofhardware and programming (for example, programmable instructions) toimplement one or more functionalities of the processing engine(s) 208.In examples described herein, such combinations of hardware andprogramming may be implemented in several different ways. For example,the programming for the processing engine(s)208 may be processorexecutable instructions saved on a non-transitory machine-readablestorage medium and the hardware for the processing engine(s) 208 maycomprise a processing resource (for example, one or more processors), toexecute such instructions. In the present examples, the machine-readablestorage medium may store instructions that, when executed by theprocessing resource, implement the processing engine(s) 208. In suchexamples, the processing unit 106 may comprise the machine-readablestorage medium storing the instructions and the processing resource toexecute the instructions, or the machine-readable storage medium may beseparate but accessible to the processing unit 106 and the processingresource. In other examples, the processing engine(s) 208 may beimplemented by electronic circuitry.

The database 210 may comprise data that is either saved or generated asa result of functionalities implemented by any of the components of theprocessing engine(s) 208.

In an exemplary embodiment, the processing engine(s) 208 may comprise areal-world information processing engine 212, context determinationengine 214, a virtual object modelling engine 216, and other engine(s)218.

It would be appreciated that engines being described are only exemplaryengines and any other engines or sub-engines may be included as part ofthe system 100 or the processing unit 106. These engines too may bemerged or divided into super-engines or sub-engines as may beconfigured.

In an aspect, one or more pre-processors of an input unit operativelycoupled with the processing unit 106 may perform pre-processing of rawdata captured by the sensor unit 102 to receive input signals byreal-world information processing engine 212.

Real-World Information Processing Engine 212

According to an embodiment, the real-world information processing engine212 receives values of the parameters from the input unit and creates a3D textual matrix of sensor representation of a real environment worldbased on the parameters. The real-world information processing engine212 creates the 3D matrix in real-time. Those skilled in the art wouldappreciate that unlike the existing methodologies, the real-worldinformation processing engine 212 creates the 3D textual matrix ofsensor representation of the real world, rather than reconstructing thereal world in plane 3D format. The created 3D matrix is in a textualformat and created using inputs received from various sensors of theinput unit (e.g., camera, accelerometer, gyroscope, etc.). An exemplarytechnique for determination of the textual information is describedbelow with reference to FIG. 3 .

Those skilled in the art would further appreciate that embodiments ofthe present invention aim to enable creating a context for virtualobjects and rendering augmented reality experience faster, without anyframe drop. Therefore, the real-world information processing engine 212can create textual information that is reusable in the three-dimensionalmatrix in such a way where if a user holding or wearing the computingdevice (incorporating a camera and the processing unit 106) traverses inreal world, the real-world information processing engine 212 maps sensorinformation in each cell (e.g. as camera moves from one cell toanother). If the user revisits a cell or remains in one, no sensor dataneeds to be processed by the real-world information processing engine212. Therefore, only when the user moves from one cell to anotherunvisited cell in the three-dimensional matrix, the sensorrepresentation of the real world is re-created in the context by storingtextual data in the matrix. Thereby, the processing capacity of thedevice is improved, which enables the device to show much higherfidelity content in augmented reality environment. Also, since the datais saved in textual format, rather than 3D digital format, storagerequirement of the system is also minimized. Those skilled in the artwould appreciate that, storing data in text format rather than 3D formathelps in keeping storage space free. Hence, read-write processes utilizeless bandwidth, thereby, keeping the system free from clogging. Further,ability to reuse previously collected data leads to the sensor data notbe collected for every frame. An exemplary technique for reusing thetextual information is described below with reference to FIG. 6 .

Context Determination Engine 214

According to an embodiment, the context determination engine 214determines a context of a specific virtual object selected from variousvirtual objects. The specific virtual object can be selected from alibrary of virtual objects maintained in database 210. The context ofthe specific virtual object is determined with respect to the real-worldenvironment based on the 3D textual matrix. Those skilled in the artwould appreciate that the context of the specific virtual object isdetermined to bring meaning to an otherwise meaningless augmentedreality environment (also referred to as virtual world, hereinafter).The camera or an image sensor captures a real-world scene to view thereal world on the computing device. Additionally, the contextdetermination engine 214 determines various factors such as surfaces,textures, light, geometries, sounds and the like. Based on the factors,each data point in the real-world scene can be treated differently andcan be used to place virtual elements in “context”.

Virtual Object Modelling Engine 216

According to an embodiment, each virtual object saved in the database210 can be divided into a plurality of virtual parts and sub-parts. Forexample, if the virtual object is a car, the car can be divided intoparts such as wheels, roof, bonnet, tailgate, doors, lights and carbody. These parts can further be divided into sub-parts, e.g. a wheelcan be divided into rim, tyre and disk-brake, a roof can be divided intobody and sunroof, where the body can further be divided into plastic,metal and chrome. Each virtual part or sub-part can be independentlymodelled, textured, compressed, and loaded as an independent virtualunit. The virtual object modelling engine 216 models the specificvirtual object based on the context to place the specific virtual objectin the augmented reality environment. In an example, every virtualobject (or parts or sub-parts) can behave like a real-world element oncethe object is placed in the augmented reality environment. All furthervirtual objects can use the object to determine new respective contextfor placement in the augmented reality environment.

Those skilled in the art would appreciate that as the context of allvirtual parts or sub-parts of the virtual object can be different andinter-dependent, the context determination engine 214 along with thevirtual object modelling engine 216 can enable the virtual parts orsub-parts to interplay with each other and the real-world environment.Moreover, such interplay can also be facilitated if more than onevirtual object is present. The context can be calculated considering theplacement and augmentation of multiple virtual objects, as the contextis not only considered from the real world, but is also calculated basedon the presence of multiple virtual objects in the scene. Therefore,embodiments of the present disclosure provide a platform-independentcontextual real-time augmented reality system that enables interactionand interplay between real-world and virtual elements (includinginterplay between various virtual objects themselves). Exemplary effectsof determination of context and modelling of virtual objects isdescribed below with reference to FIG. 4 and FIGS. 5A-C.

Those skilled in the art would appreciate that embodiments of thepresent disclosure enable independent modelling of virtual objects ortheir parts or sub-parts, where virtual objects are broken into partsand subparts and saved in the database 210 separately. Therefore, thereis an opportunity to download and load the virtual objects, parts orsub-parts, of the model according to the requirement. Storage ofinformation of the virtual object in such disintegrated manner can helpto reduce loading time by reducing need to download complete object atonce. These virtual objects can follow a specific hierarchy, which canalso help in modularity within each virtual object.

FIG. 3 illustrates exemplary representations of determination of textualinformation of the three-dimensional matrix in accordance with anembodiment of the present disclosure.

Referring to FIG. 3 , in an example, the textual information can containcontextual information of the real-world environment such as surfaceestimation 302, light estimation 304, motion estimation 306, environmentestimation 308, space estimation 310, sound estimation 312, sizeestimation 314, direction of light estimation 316 and physics estimation318. During surface estimation 302, the system 100 can determinesurfaces such as floor (horizontal), ceramic walls (vertical), metalsurface of car (horizontal and vertical), etc. Surfaces can also referto horizontal, vertical or tilted planar information of the real world.During light estimation 304, the system 102 can determine lightluminosity (e.g. 3.846×1026 watts). Light information can refer toluminosity, intensity, colour, spread, radius, perimeter, area, density,scattering, shadows, etc. of the real world. During motion estimation306, the system 100 can determine vectors for each element in the realworld or virtual world. Motion can also refer to velocity, displacement,acceleration, height, rotation, frame of reference, etc. pertaining tothe computing device. During environment estimation 308, the system 100can determine environment/time of the day e.g. beach (mid-day), city(Mid-day), mountains (morning), etc. Environment can detect surroundingsof the real-world including objects, weather, time of day, tint,geography, etc. During space estimation 310, the system 100 candetermine sub-section of a grid in the augmented reality where there isno existence of real-world and virtual world objects. During soundestimation 312, the system 100 determines sound sources in real worldand virtual world. Space can be calculated of available area taking intoaccount real world and virtual world elements in all three axes. Duringsize estimation 314, the system 100 determines sizes (e.g. length,breadth and height) for all real world and virtual world elements.During direction of light estimation 316, the system 100 can determinethree-dimensional rotational values for every light source along withdirection and propagation of sources of light. During physics estimation318, the system 100 can determine buoyancy of each surface of real worldand virtual world objects.

FIG. 4 illustrates exemplary representations of working of the augmentedreality system in accordance with an embodiment of the presentdisclosure.

According to an example, representation 402 illustrates a real-worldthat is visible to a user. Representation 404 illustrates capturing ofthe real-world scene (e.g. camera feed) through image sensor/camera of acomputing (e.g. mobile) device. Representation 406 illustrates thedetermination of various parameters of the real-world to perform variousestimations, e.g. surface estimation 302, light estimation 304, motionestimation 306, environment estimation 308, space estimation 310, soundestimation 312, size estimation 314, direction of light estimation 316and physics estimation 318. Representation 408 illustrates virtualelements A and B that are required to be placed in an augmented realityenvironment in the computing device. Representation 410 illustratesmodelling and placement of the virtual elements A and B along with realworld elements in the augmented reality environment and representation412 illustrated the integration of the augmented reality environmentwith the camera feed.

FIGS. 5A-C illustrate exemplary representations indicating an effect ofchange in parameters on the augmented reality environment in accordancewith an embodiment of the present disclosure.

Various embodiments of the present disclosure provide ability of twovirtual elements (e.g. A and B) to affect each other for modelling andplacement in the augmented reality environment to be presented on acomputing device. According to an example, the interplay can exist inthe form of (i) reflections and (ii) surfaces.

In context of (i) reflections, virtual elements A and B upon changinglocations can affect parameters of each other and respectivereflections. Referring to FIG. 5A, as illustrated in representation 502,reflections can be calculated independently for each A and B. The car,B, can be reflecting the trees of real world, from the front and bike,A, from the back. The bike, A, can have independent reflectioncalculations based on flowers, of the real world from the back, and thecar, B, from the front. According to representation 504, the positionsof A and B are swapped and thus, the reflections can be re-calculatedfor both A and B. Those skilled in the art would appreciate thatthree-dimensional cube maps can be calculated by taking six directionsfrom the centre of an object and mapping out the various visible virtualworld and real-world objects. For each virtual element, the calculationscan be returned as discrete images, which can be mapped to a cube thatcan then be applied as a reflection map on shaders.

In context of (ii) surfaces, each virtual element can be assigned a setof surfaces the virtual element can reside upon and also surfaces thevirtual element contains. For example, a bike, A, can reside upon afloor. Since the system can detect surfaces in the real-worldenvironment, the system can restrict the placement of the bike, A, onthe floor. If a user tries to place the bike, A, on top of the car, B,the system will not allow the placement of the bike, A, simply becausesurface of the car, B, can be also determined to be a metallic carsurface. However, if the virtual element was a table instead of a car,B, the table can be placed only on the floor. Now, considering anothervirtual element i.e. a lamp, the lamp can be placed a lamp on the floorand also on the table because interplay between a table surface and thelamp exists. Therefore, those skilled in the art would appreciate thatthe assignment for which surface a virtual element can interact with canbe done by detecting category of the virtual element and furtherdetecting the type of real-world object through image detection/objectdetection. However, in some cases, where virtual world objects are apart of a database, the surface information can also reside in thedatabase.

Those skilled in the are would appreciate that embodiments of thepresent disclosure enable all calculations to be performed in real-timein the computing device itself. The system can perform the calculationsbased on input received from sensors of the computing device i.e. datacan be collected from real-world environment can be understood andbrought into context. Real-time can also signify that any change inaugmentation of the virtual elements corresponding to the change in thevirtual scene or change corresponding to the device's input can happenwithin a short time (e.g. 50 milliseconds) so that the user cannotperceive any lag and can interact with the augmented world smoothly.

According to an example, referring to FIG. 5B, representation 522illustrates an augmented reality environment with real elements and avirtual element B, which can be considered as a sample output.Representation 524 is another illustration of the augmented realityenvironment, where addition of light in the real world changed metadatafor virtual element VE2. Representation 526 is another illustration ofthe augmented reality environment, where camera feed and real-worldparameters has changed by translation, which in turn also changesreflections and lighting metadata required for virtual element B.Representation 528 is another illustration of the augmented realityenvironment, where change in orientation of the computing device leadsto change in visible real world in the camera feed. Representation 530is another illustration of the augmented reality environment, where anychange in the environment causes change in reflection (light) andsurface metadata. Representation 532 is another illustration of theaugmented reality environment, where addition of a virtual element Aalso leads to determination of new environment and calculation of newparameters.

Referring to FIG. 5C, representation 542 illustrates an augmentedreality environment with real elements and two virtual elements A and B,which can be considered as a sample output. Representations 544A and544B illustrate independent change in properties of virtual elements Aand B while representations 546A and 546B illustrate change in multipleproperties of the two virtual elements A and B occur simultaneously.

Therefore, in light of representations of FIGS. 5A-C, those skilled inthe art would appreciate that embodiments of the present disclosure leadto an accurate context being created for placement and modelling of thevirtual elements in augmented reality environment in the computingdevice.

FIG. 6 illustrates formation of a three-dimensional grid in accordancewith an embodiment of the present disclosure.

According to an embodiment, the system 100 creates a three-dimensional(3D) grid i.e. a spatial structuring of the real world of the user anduser's environment. The 3D grid can be created in X, Y, Z-axis, whereeach unit is a cell such that information of each cell can be saved inthe 3D textual matrix. The grid can be bounded by detected volume in thereal-world environment, where larger the area, greater can be size ofthe 3D grid. Each cell in the grid can be bounded by least count of thesensors in the computing device. In one example:

Minimum possible cell side=1/2*(Least count of theaccelerometer)*(1/f)*(1/f)

-   -   where, f is frequency of accelerometer inputs

Therefore, smallest possible cell, where parameters (e.g. surface,motion, light, camera feed and environment) are constant, under threeconditions:

-   -   (i) Camera orientation is kept constant    -   (ii) Virtual Elements are kept constant    -   (iii) Real-world inputs are kept constant

A cell of the grid can hold all the metadata pertaining to augmentedreality environment to create the 3D grid. The purpose of such metadatacomes to light when the user travels between cells. Referring torepresentations 602, 604 and 606 of FIG. 6 , a path of a user in XYZcoordinate system can be noted. Initially, the user is at cell U1,therefore, all the parameters can be calculated with respect to cell U1and the textual data can be saved in a 3D matrix. The system can performthese calculations only once and not in every frame. When the user thenmoves to next cell, all metadata calculations can be done pertaining tothe new cell and saved in the 3D matrix. Similarly, considering finaldestination of the user in cell Uf, all metadata calculations can be ineach newly visited cell and the data can be saved in the 3D matrix. Inan example, 3D matrix size can be calculated as:

3D matrix size=Available ram/meta data in one cell

Once the data is saved with respect to various cells, if the user againarrives at a previously visited cell (e.g. Up), the system may notperform the calculations again and neither do calculations need tohappen every frame. The 3D matrix can be searched for the particularcell, and the saved metadata can be reused. Therefore, for fasterimplementation the virtual world again may not be required to becreated.

FIG. 7 is a flow diagram 700 illustrating a method for viewing interplaybetween real-world elements and virtual objects of an augmented realityenvironment in accordance with an embodiment of the present disclosure.

In an aspect, the proposed method may be described in general context ofcomputer executable instructions. Generally, computer executableinstructions can include routines, programs, objects, components, datastructures, procedures, modules, functions, etc., that performparticular functions or implement particular abstract data types. Themethod can also be practiced in a distributed computing environmentwhere functions are performed by remote processing devices that arelinked through a communications network. In a distributed computingenvironment, computer executable instructions may be located in bothlocal and remote computer storage media, including memory storagedevices.

The order in which the method as described is not intended to beconstrued as a limitation, and any number of the described method blocksmay be combined in any order to implement the method or alternatemethods. Additionally, individual blocks may be deleted from the methodwithout departing from the spirit and scope of the subject matterdescribed herein. Furthermore, the method may be implemented in anysuitable hardware, software, firmware, or combination thereof. However,for ease of explanation, in the embodiments described below, the methodmay be considered to be implemented in the above described system.

In an aspect, present disclosure elaborates upon a method for viewinginterplay between real-world elements and virtual objects of anaugmented reality environment, carried out according to instructionssaved in a computer device. The method comprises, at block 702,receiving values of one or more parameters of one or more real-worldelements of the augmented reality environment, from the input unit, theinput unit comprising one or more one or more sensors coupled with thecomputing device. The method comprises at block 704, creating a 3Dtextual matrix of sensor representation of a real environment worldbased on the one or more parameters. The method further comprises atblock 706, determining a context of a specific virtual object of the oneor more virtual objects with respect to the real-world environment basedon the 3D textual matrix, and at block 708, modelling the specificvirtual object based on the context to place said specific virtualobject in the augmented reality environment.

FIGS. 8A-B are a flow diagrams 800 and 850 illustrating exemplaryworking of the augmented reality system in accordance with an embodimentof the present disclosure.

Referring exemplary flow diagram 800, the process can be initiated atblock 802, where a camera of a computing device can be initialized andat block 804, where the camera generates a camera feed to capture realworld scenario on an augmented reality environment (e.g. virtualenvironment). At block 804, sensor data is generated from a plurality ofsensors to create a 3D matrix, at block 808. At block 810, theinformation of the 3D matrix is saved in a textual format. At block 812,context for a virtual object is created such that based on the contextat block 814, the virtual object is added to the augmented realityenvironment. Those skilled in the art would appreciate that each time avirtual object is selected for the augmented reality environment, a newcontext is created based on real world elements and already existingvirtual objects. In case there is a change in hardware orientation orreal-world parameters, the change is determined at block 816. Based onthe determination, either a new context can be created at block 812, orif the user had already visited the changed orientation before, thesystem can revisit the unchanged orientation at block 818 so thatcontext is recreated without processing at block 820.

Referring exemplary flow diagram 850, the process can be initiated atblock 852 by receiving camera feed at the computing device. Based oncamera feed and sensor information, at block 854, real-world lightinginformation can be calculated. At block 856, it can be determinedwhether lighting is sufficient or not. In case the lighting in notsufficient, at block 864, the system can recommend the user to have aproper lighting condition. In case the lighting is sufficient, at block858, the system can determine surface area information. At block 860,the scene can be modified according to available space such that, atblock 862, the scene can be rendered over the camera feed in theaugmented reality environment.

FIG. 9 illustrates an exemplary computer system 900 in which or withwhich embodiments of the present disclosure can be utilized inaccordance with embodiments of the present disclosure.

As shown in FIG. 9 , computer system 900 includes an external storagedevice 910, a bus 920, a main memory 930, a read only memory 940, a massstorage device 950, communication port 960, and a processor 970. Aperson skilled in the art will appreciate that computer system mayinclude more than one processor and communication ports. Examples ofprocessor 970 include, but are not limited to, an Intel® Itanium® orItanium 2 processor(s), or AMD® Opteron® or Athlon MP® processor(s),Motorola® lines of processors, FortiSOC™ system on a chip processors orother future processors. Processor 970 may include various enginesassociated with embodiments of the present invention. Communication port960 can be any of an RS-232 port for use with a modem-based dialupconnection, a 10/100 Ethernet port, a Gigabit or 10 Gigabit port usingcopper or fiber, a serial port, a parallel port, or other existing orfuture ports. Communication port 960 may be chosen depending on anetwork, such a Local Area Network (LAN), Wide Area Network (WAN), orany network to which computer system connects.

Memory 930 can be Random Access Memory (RAM), or any other dynamicstorage device commonly known in the art. Read only memory 940 can beany static storage device(s) e.g., but not limited to, a ProgrammableRead Only Memory (PROM) chips for storing static information e.g.,start-up or BIOS instructions for processor 970. Mass storage 950 may beany current or future mass storage solution, which can be used to storeinformation and/or instructions. Exemplary mass storage solutionsinclude, but are not limited to, Parallel Advanced Technology Attachment(PATA) or Serial Advanced Technology Attachment (SATA) hard disk drivesor solid-state drives (internal or external, e.g., having UniversalSerial Bus (USB) and/or Firewire interfaces), e.g. those available fromSeagate (e.g., the Seagate Barracuda 7102 family) or Hitachi (e.g., theHitachi Deskstar 7K1000), one or more optical discs, Redundant Array ofIndependent Disks (RAID) storage, e.g. an array of disks (e.g., SATAarrays), available from various vendors including Dot Hill SystemsCorp., LaCie, Nexsan Technologies, Inc. and Enhance Technology, Inc.

Bus 920 communicatively couples processor(s) 970 with the other memory,storage and communication blocks. Bus 920 can be, e.g. a PeripheralComponent Interconnect (PCI)/PCI Extended (PCI-X) bus, Small ComputerSystem Interface (SCSI), USB or the like, for connecting expansioncards, drives and other subsystems as well as other buses, such a frontside bus (FSB), which connects processor 970 to software system.

Optionally, operator and administrative interfaces, e.g. a display,keyboard, and a cursor control device, may also be coupled to bus 920 tosupport direct operator interaction with computer system. Other operatorand administrative interfaces can be provided through networkconnections connected through communication port 960. External storagedevice 910 can be any kind of external hard-drives, floppy drives,IOMEGA® Zip Drives, Compact Disc-Read Only Memory (CD-ROM), CompactDisc-Re-Writable (CD-RW), Digital Video Disk-Read Only Memory (DVD-ROM).Components described above are meant only to exemplify variouspossibilities. In no way should the aforementioned exemplary computersystem limit the scope of the present disclosure.

Those skilled in the art would appreciate that various embodimentsdisclosed herein provide systems and methods for viewing interplaybetween real-world elements and virtual objects of an augmented realityenvironment. Various embodiments render augmented reality experiencefaster (without any frame drop), and minimizes storage requirement,which keeps the processor free from clogging and enhances processingspeed. Advantageously, the systems and methods of the present disclosureare platform-independent. Further, various embodiments represent textualdata in a three-dimensional grid with plurality of cells, theinformation pertaining to each cell requires minimum processing inputs,which makes hardware of the system free from clogging and focuses onprocessing for tracking and rendering graphical elements of virtualobjects. Further, as information pertaining to each cell requiresminimum processing inputs, power utilization (e.g. battery utilization)of the system becomes very profound.

Moreover, in interpreting both the specification and the claims, allterms should be interpreted in the broadest possible manner consistentwith the context. In particular, the terms “comprises” and “comprising”should be interpreted as referring to elements, components, or steps ina non-exclusive manner, indicating that the referenced elements,components, or steps may be present, or utilized, or combined with otherelements, components, or steps that are not expressly referenced. Wherethe specification claims refer to at least one of something selectedfrom the group consisting of A, B, C . . . and N, the text should beinterpreted as requiring only one element from the group, not A plus N,or B plus N, etc.

While some embodiments of the present disclosure have been illustratedand described, those are completely exemplary in nature. The disclosureis not limited to the embodiments as elaborated herein only and it wouldbe apparent to those skilled in the art that numerous modificationsbesides those already described are possible without departing from theinventive concepts herein. All such modifications, changes, variations,substitutions, and equivalents are completely within the scope of thepresent disclosure. The inventive subject matter, therefore, is not tobe restricted except in the spirit of the appended claims.

1. A system implemented in a computing device for viewing interplaybetween one or more real-world elements and one or more virtual objectsof an augmented reality environment, said system comprising: an inputunit comprising one or more sensors coupled with the computing device,wherein the one or more sensors capture one or more parameters of theone or more real-world elements of the augmented reality environment; aprocessing unit comprising a processor coupled with a memory, the memorystoring instructions executable by the processor to: receiving values ofthe one or more parameters from the input unit and creating athree-dimensional textual matrix of sensor representation of a realenvironment world based on the one or more parameters; determining acontext of a specific virtual object of the one or more virtual objectswith respect to the real-world environment based on thethree-dimensional textual matrix; and modelling the specific virtualobject based on the context to place said specific virtual object in theaugmented reality environment, wherein the context of the specificvirtual object is determined based on the context of one or more othervirtual objects, wherein the Processing unit creates a three-dimensionalgrid indicating spatial structuring of a user and the user'senvironment, wherein the three-dimensional grid includes a Plurality ofcells, each cell in the grid is being bound by least count of thesensors in the computing device.
 2. The system of claim 1, wherein theone or more parameters include information pertaining to any or acombination of light, sound, surface, size, direction of light andphysics of the one or more real-world elements.
 3. The system of claim1, wherein the one or more sensors include any or a combination of animage sensor, a camera, an accelerometer, a sound sensor and agyroscope.
 4. The system of claim 1, wherein the three-dimensionaltextual matrix includes information pertaining to any or a combinationof space estimation, light estimation, sound estimation, surfaceestimation, environment estimation, size estimation, direction of lightestimation, and physics estimation with respect to the one or morereal-world elements.
 5. (canceled)
 6. The system of claim 1, whereineach virtual object of the one or more virtual objects is divided into aplurality of parts and subparts.
 7. The system of claim 1, whereininformation of the three-dimensional textual matrix is reusable. 8.(canceled)
 9. The system of claim 1, wherein information pertaining toeach cell of the plurality of cells is saved in the three-dimensionaltextual matrix.
 10. A method for viewing interplay between one or morereal-world elements and one or more virtual objects of an augmentedreality environment, carried out according to instructions saved in acomputer device, comprising: receiving values of one or more parametersof the one or more real-world elements of the augmented realityenvironment, from an input unit, the input unit comprising one or moresensors coupled with the computing device; creating a three-dimensionaltextual matrix of sensor representation of a real environment worldbased on the one or more parameters; determining a context of a specificvirtual object of the one or more virtual objects with respect to thereal-world environment based on the three-dimensional textual matrix;and modelling the specific virtual object based on the context to placesaid specific virtual object in the augmented reality environment,wherein the context of the specific virtual object is determined basedon the context of one or more other virtual objects, wherein theprocessing unit creates a three-dimensional grid indicating spatialstructuring of a user and the user's environment, wherein thethree-dimensional grid includes a plurality of cells, each cell in thegrid is being bound by least count of the sensors in the computingdevice.
 11. The method of claim 10, wherein the three-dimensionaltextual matrix includes information pertaining to any or a combinationof space estimation, light estimation, sound estimation, surfaceestimation, environment estimation, size estimation, direction of lightestimation, and physics estimation with respect to the one or morereal-world elements.
 12. (canceled)
 13. The method of claim 10, whereineach virtual object of the one or more virtual objects is divided into aplurality of parts and subparts.
 14. (canceled)
 15. The method of claim10, wherein information pertaining to each cell of the plurality ofcells is saved in the three-dimensional textual matrix.
 16. A computerprogram product, comprising: a non-transitory computer readable storagemedium comprising computer readable program code embodied in the mediumthat is executable by one or more processors of a computing device toview interplay between one or more real-world elements and one or morevirtual objects of an augmented reality environment, wherein the one ormore processors perform: receiving values of one or more parameters ofthe one or more real-world elements of the augmented realityenvironment, from an input unit, the input unit comprising one or moreone or more sensors coupled with the computing device; creating athree-dimensional textual matrix of sensor representation of a realenvironment world based on the one or more parameters; determining acontext of a specific virtual object of the one or more virtual objectswith respect to the real-world environment based on thethree-dimensional textual matrix; and modelling the specific virtualobject based on the context to place said specific virtual object in theaugmented reality environment, wherein the context of the specificvirtual object is determined based on the context of one or more othervirtual objects, wherein the processing unit creates a three-dimensionalgrid indicating spatial structuring of a user and the user'senvironment, wherein the three-dimensional grid includes a plurality ofcells, each cell in the grid is being bound by least count of thesensors in the computing device.
 17. The computer program product ofclaim 16, wherein the three-dimensional textual matrix includesinformation pertaining to any or a combination of space estimation,light estimation, sound estimation, surface estimation, environmentestimation, size estimation, direction of light estimation, and physicsestimation with respect to the one or more real-world elements. 18.(canceled)
 19. (canceled)
 20. The computer program product of claim 16,wherein information pertaining to each cell of the plurality of cells issaved in the three-dimensional textual matrix.
 21. The system of claim1, wherein the least count of the sensors in the computing device isdefined by the following formula:Minimum possible cell side=1/2*(Least count of theaccelerometer)*(1/f)*(1/f) where, f is frequency of accelerometer inputs